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OBJECTIVES OF THE STUDY 


A lthough the number of honors programs and colleges has grown dra¬ 
matically over the last twenty years to over a thousand programs nation¬ 
ally (Member List 2010, National Collegiate Honors Council), little has been 
done to develop consistency of standards in honors. In the interest of seeking 
consistency, I designed a research study to provide initial insights into assess¬ 
able measures that a panel of honors administrators agree are meaningful for 
use in honors program reviews. The study analyzed the content validity of 
a set of empirically and conceptually derived survey items to facilitate the 
development of an instrument to assess and review honors programs. The 
primary research question guiding the study was framed to collect data on 
the professional judgments of a group of experts regarding the relevance and 
meaningfulness of the proposed survey items for the purpose of conducting 
honors program reviews. 
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To address the research question, it was necessary to draw upon the theo¬ 
retical and practical experience of a panel of experts. Given the impassioned 
discussion of honors certification and assessment within the national honors 
education community during the time of this study the selected method also 
needed to offer anonymity to the participants. The Delphi technique provided 
a means of establishing a panel of experts who remained anonymous through¬ 
out the study even to one another, thus increasing the likelihood of dissent¬ 
ing opinions being expressed. According to Lang, “the Delphi was designed 
to optimize the use of group opinion whilst minimizing the adverse qualities 
of interacting groups” (3). The main purpose of the study was the production 
of an instrument generated through consensus. The successive iterations and 
controlled feedback that are characteristic of the Delphi method ensured that 
consensus or stability in panel members’ responses was reached. 

CONCEPTUAL FRAMEWORK 

The results of a review of relevant literature on honors assessment and 
higher education certification, as well as the Basic Characteristics established 
by the National Collegiate Honors Council, were used to guide instrument 
development. As illustrated in Figure 1, the conceptual framework for the 
study took into account the panel’s expert knowledge and administrative 
work experience in honors education. 

METHODS 

A review of the relevant literature led to construction of the initial instru¬ 
ment items. The literature review focused on assessment of honors education 
as well as certification of similar or related programs. The following sources 
provided the foundation for constructing the instrument items: the NCHC’s 
Basic Characteristics of Fully Developed Honors Programs and Colleges; the 
nine characteristics identified by Tallent-Runnels et.al., as emerging most 
often in literature on K-12 gifted programs; and the criteria developed by the 
Council for the Advancement of Standards in Higher Education (CAS) for 
College Honor Societies, Undergraduate Research Programs, and Service- 
Learning Programs. 

Following collection of demographic information, a survey consisting of 
215 quantitative instrument items was distributed to respondents, who were 
asked to rate the relevance of each item for its inclusion in an instrument to 
conduct honors program reviews. The conceptual basis for the instrument 
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was shared with the expert panel members in the introductory email that was 
sent to potential participants. 

A 4-point rating scale was used (see Figure 2) because it offered respon¬ 
dents no neutral mid-point, thereby reducing problematic analysis (Lynn). 
The 4-point rating scale also allowed me to collapse relevance ratings into 

Figure 1: The Research Design for the Study 
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Figure 2: Example Quantitative Rating Scale Used for the 
Content Validity of Items 
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dichotomous categories for analysis (e.g., a 1 or 2 rating collapsed into a cate¬ 
gory labeled “less relevant” and a 3 or 4 rating collapsed into a “more relevant” 
category). Alternatively the ratings could be treated as a Likert-type scale for 
analysis using descriptive statistics. 

Selection of participants for the study was consistent with qualitative 
methodology. Expert panel members were selected using the purposive 
sampling approach based on their expertise in honors assessment, honors 
research, and/or national leadership related to honors education. Twenty- 
nine NCHC-recognized site visitors, twenty members of the NCHC Assess¬ 
ment and Evaluation Committee, twenty members of the Research Commit¬ 
tee, and the Board of Directors were invited to participate. There was some 
overlap between these groups, but the target goal was to recruit approximately 
seventy experts. 

DATA SOURCES 

I sent each panel member an initial as well as follow-up email that intro¬ 
duced the topic, explained the nature of the Delphi method being used as well 
as the conceptual basis for the instrument, and included a link to an online 
survey. Participants were asked to avoid discussing their participation in the 
study in order to protect their anonymity as well as to avoid the influence of 
bias within the group. 

A concurrent nested strategy for data collection was employed. This 
strategy is characterized by the simultaneous collection of quantitative data 
and narrative comments in one phase of data collection (Creswell). In this 
instance, both quantitative and qualitative survey items were completed 
simultaneously by members of the expert panel in two consecutive rounds. 
Specifically, a series of two surveys were sent electronically to each respon¬ 
dent with instructions to complete and return the survey. Following the first 
survey, a summary of that round was prepared in the form of histograms, and 
modifications to the survey instrument were made. 

RESULTS 

The results in this paper include summaries of the quantitative analysis 
of survey responses. Of the seventy experts invited to participate, a total of 
forty completed at least one of the two rounds, for a total response rate of 
57%. The guiding research question for the study proposed to examine the 
level of agreement among experts about the relevance of a set of survey items 
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designed to create an instrument to conduct honors program reviews. The 
descriptive statistics analyzed included frequency raw percentile agreement, 
measures of dispersion (e.g., standard deviation and range), mean, median, 
mode, and test statistics for normality and skewness (see Table 1; all Tables 
can be found in the Appendix). 

Ten of the 116 items showed excellent agreement among experts, with all 
ratings (excluding missing values) marked as either relevant or very relevant. 
Measures of central tendency and dispersion for each item are presented in 
Table 2. Missing values were excluded from data analysis on a variable-by¬ 
variable basis. 

To test for deviations from normality, the Shapiro-Wilk test (p < 0.05) 
was computed for each individual item (see Table 3). All items had signifi¬ 
cant p-values, indicating that individual item responses were not normally 
distributed. Skewness test statistics for each item, also displayed in Table 3, 
indicated that all but two items were negatively skewed (98%). These results 
suggest the importance of using the asymmetric confidence interval formulas 
proposed by Penfield and Miller and the inappropriateness of a typical sym¬ 
metric confidence interval for item mean computations. 

Inferential statistics for the content-relevance of the 116 survey items are 
presented in this section. Missing values were excluded on either a variable- 
by-variable or test-by-test basis, as appropriate, for all analyses presented in 
this section. Prior to calculating the Content Validity Index (CVl) for each 
item, inter-rater agreement was assessed to determine whether there were any 
participant outliers that should be excluded: this analysis is based on each 
panel member’s total distance from the median, known also as Judge’s Dis¬ 
tance to the Median (JDM), and was used to identify aberrant raters. Once 
JDM was calculated for each participant, the z-score was used to determine 
whether a statistically significant difference existed among the raters. After 
the elimination of aberrant participants, a total of 32 participants remained. 

The CVI (Table 5) represents the proportion of responses rated as a 3 
(relevant) or a 4 (very relevant) by the experts. Polit, Beck, and Owen sug¬ 
gested that an item’s CVI values should be 1.00 if the number of raters is 4 or 
less, 0.80 for panels of 5 or 6 members, and 0.75 for larger panels. Because I 
was attempting to reduce the number of items included in the program review 
instrument, I chose to use an even stricter CVI criterion of > 0.90. Twenty- 
one items fell below the > 0.90 criterion. 

Of the 116 items, 108 met the strict criterion of achieving a lower-bound 
confidence interval of at least 3.0 and are noted with a single dagger (f) (see 
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Table 4). As seen in the Table 4, about half (48%) of the items that met the 
lower-bound 3.0 criterion also had confidence interval lengths indicating 
that the sample size was adequate to estimate the population mean with 95% 
confidence (e.g., Type I error rate of 0.05). An additional 52% met the 3.0 
asymmetric confidence interval criterion but did not have confidence interval 
lengths equal to or less than the average length. The length for these items 
ranged from 0.44 to 0.51, which is only slightly wider than the 0.43 average 
length. 

A total of 93 items met both the CVI criterion of > 0.90 and the > 3.0 
criterion for the lower-bound asymmetric score confidence interval. Table 6 
provides a summary of the content validity test results presented in this sec¬ 
tion with check marks to indicate whether each item met the different con¬ 
tent validity criteria. 

Quantitative data analysis of survey items was used to examine content 
validity index (CVI > 0.90) and asymmetric confidence interval (Cl > 3.0) 
for all items. The first step in modifying the instrument was to examine items 
that could be eliminated. I wanted to avoid the creation of an instrument that 
might result in response fatigue and therefore used a more highly discriminat¬ 
ing CVI of > 0.90 rather than > 0.80, as is widely used. 

The first step to determine which items could be eliminated involved 
reviewing the quantitative content validity criteria. Six items failed to meet 
either criteria. Additionally, 17 items were marginal in meeting the content 
validity criteria. All 23 items were eliminated. The last step was to refine 
the instrument through any necessary reorganization of items using narra¬ 
tive comments. The end result was an instrument reduced to 93 items in 12 
dimensions with a CVI of 0.94. Table 7 presents the revised instrument items, 
including benchmark dimensions. 

SCHOLARLY SIGNIFICANCE 

The goal of this study was to develop, refine, and establish the content 
validity of a comprehensive instrument to assess honors programs for the 
purpose of conducting program reviews. This study successfully established 
the content validity of 93 items constituting a much-needed instrument to 
conduct honors program reviews. At this time, no other measures have been 
published for this purpose. 

This study represents the first systematic attempt to gain consensus from 
some of the prominent experts in the field ofhonors education, based on their 
experience and involvement in the national movement, about content validity 
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of a program review instrument. Several researchers have noted the value of 
using an expert panel, such as with the Delphi method, to establish content 
validity of an instrument (Lang; Spinelli; Tigelaar, et. al.). This study adds to 
the existing literature on content validity obtained through group consensus 
of an expert panel. Additionally, this study demonstrates the successful out¬ 
comes that can be achieved when an expert panel is thoughtfully selected and 
recruited to participate in validation of a survey or instrument. 

This instrument offers a tool that can be used by honors programs to 
determine their strengths, weaknesses, and outcomes, allowing for effective 
assessment and evaluation and thereby establishing the program’s relevance. 
Financial constraints partnered with societal pressures are increasing the 
demand for assessment and data-driven decisions at the institutional level. 
Honors programs have had no standardized way to declare their legitimacy 
or relevance, but, through data-driven assessment and a stronger system of 
program reviews, honors educators can begin taking steps to do so. 
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APPENDIX 

Tables 

Table 1: Frequency and Percent of Experts' Ratings of Survey 
Items and Missing Data 


Rating Frequency (%) 


Item 

Not 

Relevant 

Somewhat 

Relevant 

Relevant 

Very 

Relevant 

Missing 

1 

1 (2.9) 

1 (2.9) 

10 (28.6) 

22 (6.29) 

1 (2.9) 

2 

1 (2.9) 

1 (2.9) 

13(37.1) 

20(57.1) 

0(0) 

3 

1 (2.9) 

0(0) 

7(20) 

27 (77.1) 

0(0) 

4 

1 (2.9) 

2 (5.7) 

8(22) 

24 (68.6) 

0(0) 

5 

2 (5.7) 

1 (2.9) 

14 (40) 

17 (48.6) 

1 (2.9) 

6 

0(0) 

2 (5.7) 

15 (42.9) 

17 (48.6) 

1 (2.9) 

7 

0(0) 

2 (5.7) 

15 (42.9) 

17 (48.6) 

1 (2.9) 

8 

1 (2.9) 

3 (8.6) 

15 (42.9) 

15 (42.9) 

1 (2.9) 

9 

2 (5.7) 

1 (2.9) 

13(37.1) 

18(51.4) 

1 (2.9) 

10 

1 (2.9) 

0(0) 

6(17.1) 

27 (77.1) 

1 (2.9) 

11 

1 (2.9) 

5 (14.3) 

9 (25.7) 

18(51.4) 

1 (2.9) 

12 

0(0) 

1 (2.9) 

6(17.1) 

27 (77.1) 

1 (2.9) 

13 

0(0) 

2 (5.7) 

11(31.4) 

21 (60) 

1 (2.9) 

14 

1 (2.9) 

5 (14.3) 

15 (42.9) 

113(37.1) 

1 (2.9) 

15 

0(0) 

1 (2.9) 

4(11.4) 

29 (82.9) 

1 (2.9) 

16 

0(0) 

3 (8.6) 

14 (40) 

17 (48.6) 

1 (2.9) 

17 

1 (2.9) 

3 (8.6) 

11 (31.4) 

19 (54.3) 

1 (2.9) 

18 

0(0) 

1 (2.9) 

8 (22.9) 

19 (54.3) 

1 (2.9) 

19 

1 (2.9) 

2 (5.7) 

18(51.4) 

13(37.1) 

1 (2.9) 

21 

0(0) 

4(11.4) 

8 (22.9) 

22 (62.9) 

1 (1.9) 

22 

1 (2.9) 

2 (5.7) 

7(20) 

24 (68.6) 

1 (2.9) 

23 

0(0) 

3 (8.6) 

9 (25.7) 

22 (62.9) 

1 (2.9) 

24 

1 (2.9) 

2 (5.7) 

9 (25.7) 

22 (62.9) 

1 (2.9) 

25 

0(0) 

1 (2.9) 

2 (5.7) 

31 (88.6) 

1 (2.9) 

26 

2 (5.7) 

1 (2.9) 

3 (8.6) 

28 (80) 

1 (2.9) 

27 

0(0) 

0(0) 

2 (5.7) 

32 (91.4) 

1 (2.9) 
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28 

0(0) 

2 (5.7) 

10 (28.6) 

22 (62.9) 

1 (2.9) 

29 

0(0) 

7(20) 

12 (34.3) 

15 (42.9) 

1 (2.9) 

30 

0(0) 

0(0) 

11(31.4) 

23 (65.7) 

1 (1.9) 

31 

1 (2.9) 

3 (8.6) 

14 (40) 

16 (45.7) 

1 (2.9) 

32 

1 (2.9) 

2 (5.7) 

11(31.4) 

20(57.1) 

1 (2.9) 

33 

0(0) 

3 (8.6) 

6(17.1) 

25 (71.4) 

1 (2.9) 

34 

1 (2.9) 

1 (2.9) 

6(17.1) 

16 (74.3) 

1 (2.9) 

35 

1 (2.9) 

10 (28.6) 

23 (65.7) 

34 (97.1) 

1 (2.9) 

36 

2 (5.7) 

1 (2.9) 

8 (22.9) 

23 (65.7) 

1 (2.9) 

37 

1 (2.9) 

2 (5.7) 

15 (42.9) 

16 (45.7) 

1 (2.9) 

38 

0(0) 

3 (8.6) 

13(37.1) 

18(51.4) 

1 (2.9) 

39 

0(0) 

2 (5.7) 

8 (22.9) 

24 (68.6) 

1 (2.9) 

40 

2 (5.7) 

2 (5.7) 

13(37.1) 

17 (48.6) 

1 (2.9) 

41 

1 (2.9) 

3 (8.6) 

9 (25.7) 

20(57.1) 

2 (5.7) 

42 

1 (2.9) 

4(11.4) 

11(31.4) 

18(51.4) 

1 (2.9) 

43 

2 (5.7) 

0(0) 

10 (28.6) 

22 (62.9) 

1 (2.9) 

44 

1 (2.9) 

0(0) 

7(20) 

26 (74.3) 

1 (2.9) 

45 

1 (2.9) 

0(0) 

15 (42.9) 

17 (48.6) 

2 (5.7) 

46 

1 (2.9) 

3 (8.6) 

15 (42.9) 

15 (42.9) 

1 (2.9) 

47 

1 (2.9) 

4(11.4) 

14 (40) 

15 (42.9) 

1 (2.9) 

48 

1 (2.9) 

3 (8.6) 

10 (28.6) 

19 (54.3) 

2 (5.7) 

49 

1 (2.9) 

3 (8.6) 

10 (28.6) 

20(57.1) 

1 (2.9) 

50 

1 (2.9) 

2 (5.7) 

12(34.3) 

19 (54.3) 

1 (2.9) 

51 

1 (2.9) 

4(11.4) 

12 (34.3) 

17 (48.6) 

1 (2.9) 

52 

1 (2.9) 

0(0) 

12 (34.3) 

20(57.1) 

1 (2.9) 

53 

2 (5.7) 

4(11.4) 

15 (42.9) 

13(37.1) 

1 (2.9) 

54 

1 (2.9) 

2 (5.7) 

16 (45.7) 

15 (42.9) 

1 (2.9) 

55 

1 (2.9) 

0(0) 

16 (45.7) 

17 (48.6) 

1 (2.9) 

56 

1 (2.9) 

3 (8.6) 

11(31.4) 

19 (54.3) 

1 (2.9) 

57 

3 (8.6) 

0(0) 

6(17.1) 

24 (68.6) 

2 (5.7) 

58 

1 (2.9) 

2 (5.7) 

11(31.4) 

19 (54.3) 

2 (5.7) 

59 

1 (2.9) 

0(0) 

8 (22.9) 

25 (71.4) 

1 (2.9) 

60 

1 (2.9) 

7(20) 

26 (74.3) 

26 (74.3) 

1 (2.9) 

61 

3 (8.6) 

0(0) 

8 (22.9) 

22 (62.9) 

2 (5.7) 


62 
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62 

3 (8.6) 

0(0) 

5 (14.3) 

26 (74.3) 

1 (2.9) 

63 

3 (8.6) 

0(0) 

4(11.4) 

27 (77.1) 

1 (2.9) 

64 

2 (5.7) 

1 (2.9) 

12 (34.3) 

19 (54.3) 

1 (2.9) 

65 

0(0) 

2 (5.7) 

8 (22.9) 

22 (62.9) 

3 (8.6) 

66 

1 (2.9) 

3 (8.6) 

8 (22.9) 

20(57.1) 

3 (8.6) 

67 

0(0) 

2 (5.7) 

10 (28.6) 

21 (60) 

2 (5.7) 

68 

0(0) 

0(0) 

9 (25.7) 

24 (68.6) 

2 (5.7) 

69 

1 (2.9) 

1 (2.9) 

12 (34.3) 

19 (54.3) 

2 (5.7) 

70 

1 (2.9) 

3 (8.6) 

10 (28.6) 

19 (54.3) 

2 (5.7) 

71 

0(0) 

3 (8.6) 

11(31.4) 

19 (54.3) 

2 (5.7) 

72 

0(0) 

4(11.4) 

10 (28.6) 

19 (54.3) 

2 (5.7) 

73 

0(0) 

3 (8.6) 

11(31.4) 

19 (54.3) 

2 (5.7) 

74 

1 (2.9) 

2 (5.7) 

14 (40) 

16 (45.7) 

2 (5.7) 

75 

1 (2.9) 

4(11.4) 

8 (22.9) 

20(57.1) 

2 (5.7) 

76 

0(0) 

0(0) 

5 (14.3) 

28 (80) 

2 (5.7) 

77 

0(0) 

1 (2.9) 

7(20) 

25 (71.4) 

2 (5.7) 

78 

0(0) 

0(0) 

6(17.1) 

27 (77.1) 

2 (5.7) 

79 

2 (5.7) 

3 (8.6) 

9 (25.7) 

19 (54.3) 

2 (5.7) 

80 

0(0) 

2 (5.7) 

7(20) 

24 (68.6) 

2 (5.7) 

81 

1 (2.9) 

1 (2.9) 

8 (22.9) 

23 (65.) 

2 (5.7) 

82 

1 (2.9) 

3 (8.6) 

11(31.4) 

18(51.4) 

2 (5.7) 

83 

1 (2.9) 

2 (5.7) 

11(31.4) 

19 (54.3) 

2 (5.7) 

84 

0(0) 

0(0) 

4(11.4) 

29 (82.9) 

2 (5.7) 

85 

1 (2.9) 

0(0) 

9 (25.7) 

22 (62.9) 

3 (8.6) 

86 

0(0) 

2 (5.7) 

8 (22.9) 

23 (65.7) 

2 (5.7) 

87 

0(0) 

1 (2.9) 

5 (14.3) 

27 (77.1) 

2 (5.7) 

88 

0(0) 

0(0) 

7(20) 

26 (74.3) 

2 (5.7) 

89 

0(0) 

3 (8.6) 

7(20) 

24 (68.6) 

2 (5.7) 

90 

0(0) 

3 (8.6) 

11(31.4) 

19 (54.3) 

2 (5.7) 

91 

0(0) 

0(0) 

8 (22.9) 

24 (68.6) 

3 (8.6) 

92 

1 (2.9) 

2 (5.7) 

7(20) 

22 (62.9) 

3 (8.60 

93 

1 (2.9) 

2 (5.7) 

11(31.4) 

19 (54.3) 

2 (5.7) 

94 

0(0) 

2 (5.7) 

14 (40) 

17 (48.6) 

2 (5.7) 

95 

0(0) 

2 (5.7) 

13(37.1) 

18(51.4) 

2 (5.7) 
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96 

1 (2.9) 

1 (2.9) 

13(37.1) 

17 (48.6) 

3 (8.6) 

97 

0(0) 

0(0) 

3 (8.6) 

30 (85.7) 

2 (5.7) 

98 

0(0) 

3 (8.6) 

4(11.4) 

25 (71.4) 

3 (8.6) 

99 

1 (2.9) 

2 (5.7) 

16 (45.7) 

14 (40) 

2 (5.7) 

100 

1 (2.9) 

2 (5.7) 

16 (45.7) 

13(37.1) 

3 (8.6) 

101 

1 (2.9) 

1 (2.9) 

19 (54.3) 

11(31.4) 

3 (8.6) 

102 

1 (2.0) 

2 (5.7) 

12 (34.3) 

16 (45.7) 

4(11.4) 

103 

0(0) 

0(0) 

9 (25.7) 

14 (68.6) 

2 (5.7) 

104 

0(0) 

5 (14.3) 

18(51.4) 

10 (28.6) 

2 (5.7) 

105 

0(0) 

3 (8.6) 

15 (42.9) 

15 (42.9) 

2 (5.7) 

106 

0(0) 

1 (2.9) 

11(31.4) 

21 (60) 

2 (5.7) 

107 

1 (2.9) 

4(11.4) 

10 (28.6) 

18(51.4) 

2 (5.7) 

108 

1 (2.9) 

3 (8.6) 

12 (34.3) 

17 (48.6) 

2 (5.7) 

109 

1 (2.9) 

3 (8.6) 

16 (45.7) 

13 (37.1) 

2 (5.7) 

110 

0(0) 

2 (5.7) 

15 (42.9) 

16 (45.7) 

2 (5.7) 

111 

2 (5.7) 

2 (5.7) 

13(37.1) 

16 (45.7) 

2 (5.7) 

112 

3 (8.6) 

3 (8.6) 

10 (28.6) 

17 (48.6) 

2 (5.7) 

113 

1 (2.9) 

2 (5.7) 

14 (40) 

16 (45.7) 

2 (5.7) 

114 

0(0) 

5 (14.3) 

10 (28.6) 

17 (48.6) 

2 (5.7) 

115 

1 (2.9) 

1 (2.9) 

8 (22.9) 

22 (62.9) 

3 (8.6) 

116 

0(0) 

1 (2.9) 

13(37.1) 

19 (54.3) 

2 (5.7) 

Note: Percent totals do not add up to 100, c 

ue to rounding. 
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Table 2: Descriptive Statistics-Mode, Median (Mdn), Mean 
(M), Standard Deviation (SD), and Range of Survey 
Item Responses 


Item 

Mode 

Mdn 

M 

SD 

Range 

1 

4 

4 

3.56 

0.705 

3 

2 

4 

4 

3.49 

0.702 

3 

3 

4 

4 

3.71 

0.622 

3 

4 

4 

4 

3.57 

0.739 

3 

5 

4 

3.5 

3.35 

0.812 

3 

6 

4 

3.5 

3.44 

0.613 

2 

7 

4 

3.5 

3.44 

0.613 

2 

8 

3 

3 

3.29 

0.76 

3 

9 

4 

4 

3.38 

0.817 

3 

10 

4 

4 

3.74 

0.618 

3 

11 

4 

4 

3.26 

0.931 

3 

12 

4 

4 

3.76 

0.496 

2 

13 

4 

4 

3.56 

0.613 

2 

14 

3 

3 

3.18 

0.797 

3 

15 

4 

4 

3.82 

0.459 

2 

16 

4 

3.5 

3.41 

0.657 

2 

17 

4 

4 

3.41 

0.783 

3 

18 

4 

4 

3.71 

0.524 

2 

19 

3 

3 

3.26 

0.71 

3 

20 

4 

3.5 

3.32 

0.806 

3 

21 

4 

4 

3.53 

0.706 

2 

22 

4 

4 

3.59 

0.743 

3 

23 

4 

4 

3.56 

0.66 

2 

24 

4 

4 

3.53 

0.748 

3 

25 

4 

4 

3.88 

0.409 

2 

26 

4 

4 

3.68 

0.806 

3 

27 

4 

4 

3.94 

0.239 

1 

28 

4 

4 

3.59 

0.609 

2 

29 

4 

3 

3.24 

0.781 

2 

30 

4 

4 

3.68 

0.475 

1 
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31 

4 

3 

3.32 

0.768 

3 

32 

4 

4 

3.47 

0.748 

3 

33 

4 

4 

3.65 

0.646 

2 

34 

4 

4 

3.68 

0.684 

3 

35 

4 

4 

3.62 

0.652 

3 

36 

4 

4 

3.53 

0.825 

3 

37 

4 

3 

3.35 

0.734 

3 

38 

4 

4 

3.44 

0.66 

2 

39 

4 

4 

3.65 

0.597 

2 

40 

4 

3.5 

3.32 

0.843 

3 

41 

4 

4 

3.45 

0.794 

3 

42 

4 

4 

3.35 

0.812 

3 

43 

4 

4 

3.53 

0.788 

3 

44 

4 

4 

3.71 

0.629 

3 

45 

4 

4 

3.45 

0.666 

3 

46 

3 

3 

3.29 

0.76 

3 

47 

4 

3 

3.26 

0.79 

3 

48 

4 

4 

3.42 

0.792 

3 

49 

4 

4 

3.44 

0.786 

3 

50 

4 

4 

3.44 

0.746 

3 

51 

4 

3.5 

3.32 

0.806 

3 

52 

4 

4 

3.47 

0.788 

3 

53 

3 

3 

3.15 

0.857 

3 

54 

3 

3 

3.32 

0.727 

3 

55 

4 

3.5 

3.44 

0.66 

3 

56 

4 

4 

3.41 

0.783 

3 

57 

4 

4 

3.55 

0.905 

3 

58 

4 

4 

3.45 

0.754 

3 

59 

4 

4 

3.68 

0.638 

3 

60 

4 

4 

3.71 

0.629 

3 

61 

4 

4 

3.48 

0.906 

3 

62 

4 

4 

3.59 

0.892 

3 

63 

4 

4 

3.62 

0.888 

3 

64 

4 

4 

3.41 

0.821 

3 
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65 

4 

4 

3.63 

0.609 

2 

66 

4 

4 

3.47 

0.803 

3 

67 

4 

4 

3.58 

0.614 

2 

68 

4 

4 

3.73 

0.452 

1 

69 

4 

4 

3.48 

0.712 

3 

70 

4 

4 

3.42 

0.792 

3 

71 

4 

4 

3.48 

0.667 

2 

72 

4 

4 

3.45 

0.711 

2 

73 

4 

4 

3.48 

0.667 

2 

74 

4 

3 

3.36 

0.742 

3 

75 

4 

4 

3.42 

0.83 

3 

76 

4 

4 

3.85 

0.364 

1 

77 

4 

4 

3.73 

0.517 

2 

78 

4 

4 

3.82 

0.392 

1 

79 

4 

4 

3.36 

0.895 

3 

80 

4 

4 

3.67 

0.595 

2 

81 

4 

4 

3.61 

0.704 

3 

82 

4 

4 

3.39 

0.788 

3 

83 

4 

4 

3.45 

0.754 

3 

84 

4 

4 

3.88 

0.331 

1 

85 

4 

4 

3.62 

0.66 

3 

88 

4 

4 

3.79 

0.415 

1 

91 

4 

4 

3.75 

0.44 

1 

92 

4 

4 

3.56 

0.759 

3 

93 

4 

4 

3.45 

0.754 

3 

94 

4 

4 

3.45 

0.617 

2 

95 

4 

4 

3.48 

0.619 

2 

96 

4 

4 

3.44 

0.716 

3 

97 

4 

4 

3.91 

0.292 

1 

98 

4 

4 

3.69 

0.644 

2 

99 

3 

3 

3.3 

0.728 

3 

100 

3 

3 

3.28 

0.729 

3 

101 

3 

3 

3.25 

0.672 

3 

102 

4 

4 

3.39 

0.761 

3 
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103 

4 

4 

3.73 

0.452 

1 

104 

3 

3 

3.15 

0.667 

2 

105 

3 

3 

3.36 

0.653 

2 

106 

4 

4 

3.61 

0.556 

2 

107 

4 

4 

3.36 

0.822 

3 

108 

4 

4 

3.36 

0.783 

3 

109 

3 

3 

3.24 

0.751 

3 

110 

4 

3 

3.42 

0.614 

2 

111 

4 

3 

3.3 

0.847 

3 

112 

4 

4 

3.24 

0.969 

3 

113 

4 

3 

3.36 

0.742 

3 

114 

4 

4 

3.37 

0.751 

2 

115 

4 

4 

3.59 

0.712 

3 

116 

4 

4 

3.55 

0.564 

2 

Note: *multip 

e modes exist, smallest is shown. 
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Table 3: Shapiro-Wilk (W) and Skewness Test Statistics 
for Normality 


Item 

W 

P 

Skewness 

1 

0.58 

0.00 

-1.84 

2 

0.74 

0.00 

-0.58 

3 

0.58 

0.00 

-0.95 

4 

0.67 

0.00 

-1.28 

5 

0.74 

0.00 

-1.34 

6 

0.63 

0.00 

-1.51 

7 

0.74 

0.00 

-0.89 

8 

0.78 

0.00 

-0.55 

9 

0.74 

0.00 

-1.34 

10 

0.50 

0.00 

-1.62 

11 

0.76 

0.00 

-0.95 

12 

0.52 

0.00 

-2.24 

13 

0.67 

0.00 

-1.25 

14 

0.78 

0.00 

-0.50 

15 

0.45 

0.00 

-2.74 

16 

0.74 

0.00 

-0.85 

17 

0.72 

0.00 

-1.45 

18 

0.63 

0.00 

-1.52 

19 

0.78 

0.00 

-0.25 

20 

0.82 

0.00 

-0.76 

21 

0.71 

0.00 

-1.08 

22 

0.63 

0.00 

-1.51 

23 

0.63 

0.00 

-1.51 

24 

0.74 

0.00 

-0.89 

25 

0.35 

0.00 

-3.44 

26 

0.54 

0.00 

-2.24 

27 

0.24 

0.00 

-4.47 

28 

0.72 

0.00 

-0.78 

29 

0.78 

0.00 

-0.50 

30 

0.61 

0.00 

-0.68 

31 

0.79 

0.00 

-1.10 
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32 

0.72 

0.00 

-0.78 

33 

0.63 

0.00 

-1.51 

34 

0.58 

0.00 

-1.84 

35 

0.63 

0.00 

-0.44 

36 

0.66 

0.00 

-1.86 

37 

0.78 

0.00 

-0.40 

38 

0.74 

0.00 

-0.89 

39 

0.67 

0.00 

-1.25 

40 

0.72 

0.00 

-1.45 

41 

0.77 

0.00 

-0.70 

42 

0.78 

0.00 

-0.50 

43 

0.69 

0.00 

-1.67 

44 

0.54 

0.00 

-1.25 

45 

0.64 

0.00 

0.22 

46 

0.78 

0.00 

-0.40 

47 

0.78 

0.00 

-0.40 

48 

0.74 

0.00 

-0.89 

49 

0.71 

0.00 

-1.08 

50 

0.67 

0.00 

-1.25 

51 

0.77 

0.00 

-0.70 

52 

0.62 

0.00 

-2.19 

53 

0.80 

0.00 

-1.02 

54 

0.74 

0.00 

-0.21 

55 

0.64 

0.00 

0.00 

56 

0.74 

0.00 

-0.89 

57 

0.58 

0.00 

-2.07 

58 

0.76 

0.00 

-0.71 

59 

0.54 

0.00 

-1.25 

60 

0.54 

0.00 

-1.25 

61 

0.68 

0.00 

-1.67 

62 

0.58 

0.00 

-2.07 

63 

0.58 

0.00 

-2.07 

64 

0.75 

0.00 

-1.42 

65 

0.74 

0.00 

-0.89 
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66 

0.76 

0.00 

-1.12 

67 

0.63 

0.00 

-1.52 

68 

0.54 

0.00 

-1.25 

69 

0.68 

0.00 

-1.82 

70 

0.75 

0.00 

-1.42 

71 

0.72 

0.00 

-0.78 

72 

0.77 

0.00 

-0.70 

73 

0.74 

0.00 

-0.89 

74 

0.76 

0.00 

-1.31 

75 

0.75 

0.00 

-1.32 

76 

0.43 

0.00 

-2.12 

77 

0.52 

0.00 

-2.24 

78 

0.43 

0.00 

-2.12 

79 

0.72 

0.00 

-1.13 

80 

0.63 

0.00 

-1.52 

81 

0.72 

0.00 

-1.55 

82 

0.79 

0.00 

-1.10 

83 

0.75 

0.00 

-1.32 

84 

0.50 

0.00 

-1.62 

85 

0.58 

0.00 

-2.42 

86 

0.58 

0.00 

-1.78 

87 

0.45 

0.00 

-2.74 

88 

0.50 

0.00 

-1.62 

89 

0.58 

0.00 

-1.84 

90 

0.74 

0.00 

-0.89 

91 

0.50 

0.00 

-1.62 

92 

0.66 

0.00 

-1.86 

93 

0.72 

0.00 

-1.45 

94 

0.76 

0.00 

-0.71 

95 

0.74 

0.00 

-0.89 

96 

0.75 

0.00 

-1.42 

97 

0.35 

0.00 

-2.89 

98 

0.62 

0.00 

-1.39 

99 

0.81 

0.00 

-0.95 
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100 

0.81 

0.00 

-0.95 

101 

0.76 

0.00 

-1.10 

102 

0.81 

0.00 

-0.95 

103 

0.50 

0.00 

-1.62 

104 

0.81 

0.00 

-0.19 

105 

0.77 

0.00 

-0.55 

106 

0.63 

0.00 

-1.52 

107 

0.79 

0.00 

-0.85 

108 

0.71 

0.00 

-1.02 

109 

0.83 

0.00 

-0.70 

110 

0.78 

0.00 

-0.40 

111 

0.81 

0.00 

-0.99 

112 

0.79 

0.00 

-0.90 

113 

0.80 

0.00 

-1.02 

114 

0.78 

0.00 

-0.41 

115 

0.70 

0.00 

-1.69 

116 

0.74 

0.00 

-0.58 
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Table 4: Item Means and 95% Asymmetric Score Confidence 
Intervals (Cl) 
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31 

32 

3.34 

3 . 07 f 

3.56 

0.28 

0.21 

0.49 

32 

32 

3.53 

3 . 27 f 

3.71 

0.26 

0.18 

0 . 43 + 

33 

32 

3.69 

3 . 46 f 

3.83 

0.23 

0.14 

0 . 37 + 

34 

32 

3.75 

3 . 53 t 

3.87 

0.22 

0.12 

0 . 34 + 

35 

32 

3.69 

3 . 46 + 

3.83 

0.23 

0.14 

0 . 37 + 

36 

32 

3.59 

3 . 35 + 

3.76 

0.25 

0.16 

0 . 41 + 

37 

32 

3.41 

3 . 13 + 

3.61 

0.27 

0.20 

0.47 

38 

32 

3.44 

3 . 17 + 

3.63 

0.27 

0.20 

0.47 

39 

32 

3.69 

3 . 46 + 

3.83 

0.23 

0.14 

0 . 37 + 

40 

32 

3.38 

3 . 10 + 

3.58 

0.28 

0.21 

0.48 

41 

31 

3.52 

3 . 25 + 

3.70 

0.26 

0.18 

0.45 

42 

32 

3.41 

3 . 13 + 

3.61 

0.27 

0.20 

0.47 

43 

32 

3.59 

3 . 35 + 

3.76 

0.25 

0.16 

0 . 41 + 

44 

32 

3.78 

3 . 57 + 

3.89 

0.21 

0.11 

0 . 32 + 

45 

31 

3.52 

3 . 25 + 

3.70 

0.26 

0.18 

0.45 

46 

32 

3.34 

3 . 07 + 

3.56 

0.28 

0.21 

0.49 

47 

32 

3.31 

3 . 03 + 

3.53 

0.28 

0.22 

0.50 

48 

31 

3.48 

3 . 22 + 

3.67 

0.27 

0.19 

0.46 

49 

32 

3.50 

3 . 24 + 

3.68 

0.26 

0.18 

0.45 

50 

32 

3.50 

3 . 24 + 

3.68 

0.26 

0.18 

0.45 

51 

32 

3.38 

3 . 10 + 

3.58 

0.28 

0.21 

0.48 

52 

32 

3.53 

3 . 27 + 

3.71 

0.26 

0.18 

0 . 43 + 

53 

32 

3.19 

2.90 

3.42 

0.29 

0.24 

0.53 

54 

32 

3.38 

3 . 10 + 

3.58 

0.28 

0.21 

0.48 

55 

32 

3.50 

3 . 24 + 

3.68 

0.26 

0.18 

0.45 

56 

32 

3.47 

3 . 20 + 

3.66 

0.26 

0.19 

0.46 

57 

31 

3.61 

3 . 36 + 

3.77 

0.25 

0.16 

0 . 41 + 

58 

31 

3.52 

3 . 25 + 

3.70 

0.26 

0.18 

0.45 

59 

32 

3.75 

3 . 53 + 

3.87 

0.22 

0.12 

0 . 34 + 

60 

32 

3.78 

3 . 57 + 

3.89 

0.21 

0.11 

0 . 32 + 

61 

31 

3.55 

3 . 29 + 

3.72 

0.26 

0.18 

0.44 

62 

32 

3.66 

3 . 42 + 

3.80 

0.24 

0.15 

0 . 39 + 

63 

32 

3.69 

3 . 46 + 

3.83 

0.23 

0.14 

0 . 37 + 

64 

32 

3.47 

3 . 20 + 

3.66 

0.26 

0.19 

0.46 
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31 

3.61 

3 . 36 + 

3.77 

0.25 

0.16 

0 . 41 + 

66 

31 

3.45 

3 . 18 + 

3.65 

0.27 

0.20 

0.47 

67 

32 

3.56 

3 . 31 + 

3.73 

0.25 

0.17 

0 . 42 + 

68 

32 

3.72 

3 . 49 + 

3.85 

0.22 

0.13 

0 . 36 + 

69 

32 

3.47 

3 . 20 + 

3.66 

0.26 

0.19 

0.46 

70 

32 

3.41 

3 . 13 + 

3.61 

0.27 

0.20 

0.47 

71 

32 

3.47 

3 . 20 + 

3.66 

0.26 

0.19 

0.46 

72 

32 

3.44 

3 . 17 + 

3.63 

0.27 

0.20 

0.47 

73 

32 

3.47 

3 . 20 + 

3.66 

0.26 

0.19 

0.46 

74 

32 

3.34 

3 . 07 + 

3.56 

0.28 

0.21 

0.49 

75 

32 

3.41 

3 . 13 + 

3.61 

0.27 

0.20 

0.47 

76 

32 

3.84 

3 . 65 + 

3.93 

0.19 

0.09 

0 . 28 + 

77 

32 

3.72 

3 . 49 + 

3.85 

0.22 

0.13 

0 . 36 + 

78 

32 

3.81 

3 . 61 + 

3.91 

0.20 

0.10 

0 . 30 + 

79 

32 

3.38 

3 . 10 + 

3.58 

0.28 

0.21 

0.48 

80 

32 

3.66 

3 . 42 + 

3.80 

0.24 

0.15 

0 . 39 + 

81 

32 

3.59 

3 . 35 + 

3.76 

0.25 

0.16 

0 . 41 + 

82 

32 

3.38 

3 . 10 + 

3.58 

0.28 

0.21 

0.48 

83 

32 

3.47 

3 . 20 + 

3.66 

0.26 

0.19 

0.46 

84 

32 

3.88 

3 . 69 + 

3.95 

0.18 

0.08 

0 . 26 + 

85 

31 

3.61 

3 . 36 + 

3.77 

0.25 

0.16 

0 . 41 + 

86 

32 

3.63 

3 . 38 + 

3.78 

0.24 

0.16 

0 . 40 + 

87 

32 

3.78 

3 . 57 + 

3.89 

0.21 

0.11 

0 . 32 + 

88 

32 

3.78 

3 . 57 + 

3.89 

0.21 

0.11 

0 . 32 + 

89 

32 

3.66 

3 . 42 + 

3.80 

0.24 

0.15 

0 . 39 + 

90 

32 

3.47 

3 . 20 + 

3.66 

0.26 

0.19 

0.46 

91 

31 

3.74 

3 . 52 + 

3.87 

0.22 

0.13 

0 . 35 + 

92 

31 

3.55 

3 . 29 + 

3.72 

0.26 

0.18 

0.44 

93 

32 

3.44 

3 . 17 + 

3.63 

0.27 

0.20 

0.47 

94 

32 

3.44 

3 . 17 + 

3.63 

0.27 

0.20 

0.47 

95 

32 

3.47 

3 . 20 + 

3.66 

0.26 

0.19 

0.46 

96 

31 

3.42 

3 . 14 + 

3.62 

0.28 

0.20 

0.48 

97 

32 

3.91 

3 . 74 + 

3.97 

0.17 

0.06 

0 . 23 + 

98 

31 

3.68 

3 . 44 + 

3.82 

0.24 

0.14 

0 . 38 + 
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99 

32 

3.28 

3.00f 

3.50 

0.28 

0.22 

0.51 

100 

31 

3.26 

2.97 

3.49 

0.29 

0.23 

0.52 

101 

31 

3.26 

2.97 

3.49 

0.29 

0.23 

0.52 

102 

30 

3.37 

3.08f 

3.58 

0.29 

0.21 

0.50 

103 

32 

3.72 

3.49t 

3.85 

0.22 

0.13 

0.36+ 

104 

32 

3.16 

2.87 

3.40 

0.29 

0.24 

0.53 

105 

32 

3.34 

3.07f 

3.56 

0.28 

0.21 

0.49 

106 

32 

3.59 

3.35t 

3.76 

0.25 

0.16 

0.41+ 

107 

32 

3.34 

3.07f 

3.56 

0.28 

0.21 

0.49 

108 

32 

3.38 

3.10+ 

3.58 

0.28 

0.21 

0.48 

109 

32 

3.22 

2.93 

3.45 

0.29 

0.23 

0.52 

110 

32 

3.41 

3.13+ 

3.61 

0.27 

0.20 

0.47 

111 

32 

3.28 

3.00+ 

3.50 

0.28 

0.22 

0.51 

112 

32 

3.22 

2.93 

3.45 

0.29 

0.23 

0.52 

113 

32 

3.34 

3.07+ 

3.56 

0.28 

0.21 

0.49 

114 

31 

3.35 

3.07+ 

3.57 

0.28 

0.21 

0.50 

115 

31 

3.58 

3.33+ 

3.75 

0.25 

0.17 

0.42+ 

116 

32 

3.53 

3.27+ 

3.71 

0.26 

0.18 

0.43+ 


Note: f Lower limit of confidence interval for the mean > 3.0 criterion. 


+ Length of confidence interval < 0.49 (the average length). 
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Table 5: Content Validity Index (CVI) Results and Exact 
Binomial Confidence Intervals (CIs) 


Item 

CVI 

95% Cl 

Lower Upper 

1 

93.6 

.84 

1.03 

2 

93.8 

.85 

1.03 

3 

96.9 

.91 

1.03 

4 

90.6 

.80 

1.01 

5 

90.6 

.80 

1.01 

6 

93.8 

.85 

1.03 

7 

93.8 

.85 

1.03 

8 

90.6 

.80 

1.01 

9 

90.6 

.80 

1.01 

10 

100 

— 

— 

11 

81.3 

.67 

.96 

12 

96.9 

.91 

1.03 

13 

93.8 

.85 

1.03 

14 

84.4 

.71 

.98 

15 

96.9 

.91 

1.03 

16 

90.6 

.80 

1.01 

17 

90.6 

.80 

1.01 

18 

96.9 

.91 

1.03 

19 

93.8 

.85 

1.03 

20 

84.4 

.71 

.98 

21 

90.6 

.80 

1.01 

22 

93.8 

.85 

1.03 

23 

90.6 

.80 

1.01 

24 

93.8 

.85 

1.03 

25 

96.9 

.91 

1.03 

26 

90.6 

.80 

1.01 

27 

100 

— 

— 

28 

93.8 

.85 

1.03 

29 

81.3 

.67 

.96 

30 

100 

— 

— 


77 






































Patricia Joanne Smith 


31 

90.6 

.80 

1.01 

32 

93.8 

.85 

1.03 

33 

93.8 

.85 

1.03 

34 

96.9 

.91 

1.03 

35 

100 

— 

— 

36 

93.8 

.85 

1.03 

37 

93.8 

.85 

1.03 

38 

90.6 

.80 

1.01 

39 

96.9 

.91 

1.03 

40 

90.6 

.80 

1.01 

41 

90.3 

.79 

1.01 

42 

87.5 

.75 

1.00 

43 

96.9 

.91 

1.03 

44 

100 

— 

— 

45 

100 

— 

— 

46 

90.6 

.80 

1.01 

47 

87.5 

.75 

1.00 

48 

90.3 

.79 

1.01 

49 

90.6 

.80 

1.01 

50 

93.8 

.85 

1.03 

51 

87.5 

.75 

1.00 

52 

96.9 

.91 

1.03 

53 

84.4 

.71 

.98 

54 

93.8 

.85 

1.03 

55 

100 

— 

— 

56 

90.6 

.80 

1.01 

57 

93.6 

.84 

1.03 

58 

93.6 

.84 

1.03 

59 

100 

— 

— 

60 

100 

— 

— 

61 

93.6 

.84 

1.03 

62 

93.8 

.85 

1.03 

63 

93.8 

.85 

1.03 

64 

93.8 

.85 

1.03 
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65 

93.6 

.84 

1.03 

66 

87.1 

.75 

1.00 

67 

93.8 

.85 

1.03 

68 

100 

— 

— 

69 

93.8 

.85 

1.03 

70 

87.5 

.75 

1.00 

71 

90.6 

.80 

1.01 

72 

87.5 

.75 

1.00 

73 

90.6 

.80 

1.01 

74 

90.6 

.80 

1.01 

75 

84.4 

.71 

.98 

76 

100 

— 

— 

77 

96.9 

.91 

1.03 

78 

100 

— 

— 

79 

84.4 

.71 

.98 

80 

93.8 

.85 

1.03 

81 

93.8 

.85 

1.03 

82 

87.5 

.75 

1.00 

83 

90.6 

.80 

1.01 

84 

100 

— 

— 

85 

96.8 

.90 

1.03 

86 

93.8 

.85 

1.03 

87 

96.9 

.91 

1.03 

88 

100 

— 

— 

89 

93.8 

.85 

1.03 

90 

90.6 

.80 

1.01 

91 

100 

— 

— 

92 

90.3 

.79 

1.01 

93 

90.6 

.80 

1.01 

94 

93.8 

.85 

1.03 

95 

93.8 

.85 

1.03 

96 

93.6 

.84 

1.03 

97 

100 

— 

— 

98 

90.3 

.79 

1.01 
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99 

90.6 

.80 

1.01 

100 

90.3 

.79 

1.01 

101 

93.6 

.84 

1.03 

102 

90 

.79 

1.01 

103 

100 

— 

— 

104 

84.4 

.71 

.98 

105 

90.6 

.80 

1.01 

106 

96.9 

.91 

1.03 

107 

84.4 

.71 

.98 

108 

87.5 

.75 

1.00 

109 

87.5 

.75 

1.00 

110 

93.8 

.85 

1.03 

111 

87.5 

.75 

1.00 

112 

81.3 

.67 

.96 

113 

90.6 

.80 

1.01 

114 

83.9 

.70 

.98 

115 

93.6 

.84 

1.03 

116 

96.9 

.91 

1.03 


Note: Where data is omitted (—), item is constant. 
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Table 6: Summary of Results for Content Validity Index (CVI) 
and Asymmetric Confidence Interval (Cl) Criteria 



81 








































Patricia Joanne Smith 


27 

/ 


28 

/ 

/ 

29 



30 

/ 

/ 

31 

/ 

/ 

32 

/ 

/ 

33 

/ 

/ 

34 

/ 

/ 

35 

/ 

/ 

36 

/ 

/ 

37 

/ 

/ 

38 

/ 

/ 

39 

/ 

/ 

40 

/ 

/ 

41 

/ 

/ 

42 


/ 

43 

/ 

/ 

44 

/ 

/ 

45 

/ 

/ 

46 

/ 

/ 

47 


/ 

48 

/ 

/ 

49 

/ 

/ 

50 

/ 

/ 

51 


/ 

52 

/ 

/ 

53 



54 

/ 

/ 

55 

/ 

/ 

56 

/ 

/ 

57 

/ 

/ 

58 

/ 

/ 
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59 

/ 


60 

/ 


Legal and Ethical 

61 

/ 


62 

/ 


Equity and Access 

63 

/ 


64 

/ 


Diversity 

65 

/ 


66 



Organization and Management 

67 

/ 


68 

/ 


69 

/ 


70 



71 

/ 


72 



73 

/ 


74 



75 



Student Services 

76 



77 



78 



79 



80 



81 



82 



83 



84 




83 
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Curiculum and Scholarship 

85 

/ 


86 

/ 


87 

/ 


88 

/ 


89 

/ 


90 

/ 


91 

/ 


Campus and External Relations 

92 

/ 


93 

/ 


94 

/ 


95 

/ 


96 

/ 


Financial Resources 

97 



98 



99 

/ 


100 



Technology 

101 



102 



Facilities and Equipment 

103 



104 



105 



106 



107 



108 



Assessment and Evaluation 

109 



110 



111 
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112 



113 

/ 

/ 

114 


/ 

115 

/ 

/ 

116 

/ 

/ 


Note: ^ indicates that the item met this criterion. 
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Table 7: Revised Instrument Items 


Item 

Items sorted into Benchmark Domains 

Mission and Objectives Benchmarks 

1 

HP develops, disseminates, and regularly reviews its mission and objectives. 

2 

HP implements its mission and objectives. 

3 

The mission statement is consistent with that of the host institution. 

Program Benchmarks 

4 

HP engages students in investigative and creative activity. 

5 

HP provides evidence of its impact on identified achievement of student 
learning and development outcomes. 

6 

HP is integrated into the life of the institution 

7 

HP is intentional and coherent 

8 

HP is responsive to needs of individuals, diverse and special populations, 
and relevant constituencies. 


HP Programs Include: 

9 

a) educational programming complementing the academic curriculum 

10 

b) faculty, staff, and administrators who are involved and interact with 
students. 

11 

HP creates an active learning environment supportive of scholarship and 
research 

12 

HP integrates research activities with professional and liberal education 

13 

HP promotes intellectual rigor and student intellectual growth and 
development 

14 

HP requires an appropriate report of the student’s completed work 

15 

HP encourages research that is commensurate with practice in the 
disciplines. 

16 

HP is constructed to ensure intellectual rigor 

17 

HP provides professional development and support to faculty and staff. 

18 

HP provides a locus of visible and highly reputed standards and models of 
excellence for students and faculty across the campus. 

19 

HP chooses technology that facilitates student learning and development 
and reflects current best pedagogical practices. 

Personnel Benchmarks 

20 

Dean or Director reports directly to the chief academic officer of the 
institution 
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21 

If unit head is Dean, serves as a full member of the Council of Deans if one 
exists 

22 

Dean or Director has a fulltime, 12-month appointment. 


HP Leaders: 

23 

a) articulate a vision and mission for the program 

24 

b) set goals and objectives based on the needs of the population and desired 
student learning and development outcomes 

25 

c) advocate for the program 

26 

d) promote campus environments that provide meaningful opportunities for 
student learning, development, and integration 

27 

e) advocate for representation in strategic planning initiatives at appropriate 
divisional and institutional levels 

28 

f) initiate collaborative interactions with stakeholders who have legitimate 
concerns and interests in the functional area 

29 

g) apply effective practices to educational and administrative processes 

30 

h) prescribe and model ethical behavior 

31 

i) communicate effectively 

32 

j) manage financial resources, including planning, allocation, monitoring, 
and analysis 

33 

k) manage human resource processes including, recruitment, selection, 
development, supervision, performance planning, and evaluation 

34 

1) empower professional, support, and student staff to accept leadership 
opportunities 

35 

m) encourage and support scholarly contribution to the profession 

36 

n) develop and continuously improve programs and services in response 
to the changing needs of students and other populations and the evolving 
institutional priorities 


HP Staff Members: 

37 

a) represent the institution 

38 

b) model leadership principles 

39 

c) fairly assess student performance. 

40 

HP is staffed adequately with personnel qualified to accomplish the mission 
and goals. 

41 

Procedures are in place for staff selection, training, and evaluation. 

42 

HP provides professional development opportunities to improve the 
professional competence, leadership ability, and skills of all staff members. 
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Staff Members, Student Leaders, Employees and Volunteers: 

43 

a) are carefully selected, trained, supervised, and evaluated 

44 

b) are educated on how and when to refer those in need of additional 
assistance to a qualified staff member 

45 

c) have access to a supervisor for assistance in making judgments 

46 

HP has adequate technical and support staff to accomplish the mission 

47 

HP staff members receive training on policies and procedures related to the 
use of technology to store or access student records and institutional data. 

48 

Salary levels and benefits are commensurate with those of comparable 
positions within the institution, similar institutions, and geographic area. 

49 

Position descriptions for all staff members are maintained. 

50 

Hiring and promotion practices are fair, inclusive, proactive, and non- 
discriminatory. 

51 

Regular performance planning and evaluation of staff members and leaders 
are conducted. 

52 

The honors program exercises considerable control over the selection of its 
faculty. 

53 

HP faculty are selected on basis of exceptional teaching skills, the ability to 
provide intellectual leadership and mentoring for able students, and support 
for the mission of honors education. 

Legal and Ethical Benchmarks 

54 

HP staff members comply with the institution’s human subjects research 
and other policies addressing confidentiality of research data concerning 
individuals 

55 

HP staff members ensure that funds are managed in accordance with 
established institutional accounting procedures and fiscal policies. 

Equity and Access Benchmarks 

56 

HP provides services on a fair, equitable, and non-discriminatory basis. 

57 

HP includes outreach efforts to under-represented populations in 
membership recruitment activities. 

58 

HP nurtures environments that are welcoming to and bring together persons 
of diverse backgrounds 

Organization and Management Benchmarks 

59 

The honors program exercises considerable control over its organizational 
policies. 

60 

HP is managed effectively. 
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61 

Websites are monitored to ensure currency, accuracy, appropriate references, 
and accessibility. 

62 

HP uses clear sources and channels of authority 

63 

HP uses systems of accountability and evaluation 

64 

HP uses processes for recognition and award. 

Student Services Benchmarks 

65 

HP exercises considerable control over honors recruitment and admissions, 
including the appropriate size of the incoming class. Admission to the 
honors program may be by separate application. 

66 

HP clearly specifies the requirements needed for retention and satisfactory 
completion. 

67 

Honors students are assured a voice in the governance and direction of 
the honors program. This can be achieved through a student committee 
that conducts its business with as much autonomy as possible but works in 
collaboration with the administration and faculty to maintain excellence in 
the program. 

68 

HP emphasizes active learning and participatory education by offering 
opportunities for students to participate in regional and national 
conferences, Honors Semesters, international programs, community 
service, internships, undergraduate research, and other types of experiential 
education. 

69 

HP provides priority enrollment for active honors students in recognition of 
scheduling difficulties caused by the need to satisfy both honors and major 
program (s) requirements. 

70 

Where the home university has a significant residential component, the HP 
offers substantial honors residential opportunities. 

71 

The distinction achieved by the completion of the HP requirements 
is publically announced and recorded, and methods may include 
announcement at commencement ceremonies, notations on the diploma 
and/or the student’s final transcript, or other similar actions. 

Curriculum & Scholarship Benchmarks 

72 

HP offers carefully designed educational experiences that meet the needs 
and abilities of the undergraduate students it serves. 

73 

HP curriculum, established in harmony with the mission statement, meets 
the needs of the students in the program and features special courses, 
seminars, colloquia, experiential learning opportunities, undergraduate 
research opportunities, or other independent-study options. 
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74 

HP requirements constitute a substantial portion of the participants’ 
undergraduate work, typically 20% to 25% of the total course work and 
certainly no less than 15%. 

75 

HP curriculum is designed so that honors requirements can, when 
appropriate, also satisfy general education requirements, major or 
disciplinary requirements, and preprofessional or professional training 
requirements. 

76 

HP serves as a laboratory within which faculty feel welcome to experiment 
with new subjects, approaches, and pedagogies. When proven successful, 
such efforts in curriculum and pedagogical development can serve as 
prototypes for initiatives that can become institutionalized across the 
campus. 

77 

HP curriculum offers significant course opportunities across all four years of 
study. 

78 

HP exercises considerable control over its curriculum. 

Campus and External Relations Benchmarks 

79 

The program has a standing committee or council of faculty members 
that works with the director or other administrative officer and is involved 
in honors curriculum, governance, policy, development, and evaluation 
deliberations. The composition of that group represents the colleges and/or 
departments served by the program and also elicits support for the program 
from across the campus. 

80 

Honors students are included in governance, serving on the advisory/ 
policy committee as well as constituting the group that governs the student 
association. 


HP Reaches Out to Relevant Individuals, Campus Offices, and External Agencies to: 

81 

a) establish, maintain, and promote effective relations 

82 

b) disseminate information about its own and other related programs and 
services 

83 

c) coordinate and collaborate, where appropriate, in offering programs and 
services to meet the needs of students and promote achievement of student 
learning and development outcomes. 

Financial Resources Benchmarks 

84 

HP has adequate funding to accomplish its mission and goals. 

85 

HP demonstrates fiscal responsibility and cost-effectiveness consistent with 
institutional protocols. 

86 

An analysis of expenditures, external and internal resources, and impact on 
the campus community is completed before establishing funding priorities. 
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Facilities and Equipment Benchmarks 

87 

HP has adequate facilities and equipment to support its mission and goals. 

88 

HP staff members have work space that is well-equipped, adequate in size, 
and designed to support their work and responsibilities 

89 

HP facilities are located in a suitable, preferably prominent, quarters on 
campus that provide both access for the students and a focal point for honors 
activity. 

Assessment and Evaluation Benchmarks 

90 

HP conducts regular assessment and evaluations. 

91 

Data are collected from students and other relevant constituencies. 


Results of Program Evaluations Are Used to: 

92 

a) revise and improve the program 

93 

b) identify needs and interests in shaping direction 
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